Skip to content

q4k mmq optimizations - #80

Draft
liangliangchang wants to merge 5 commits into
gfx11from
lichang.q4k-opt
Draft

q4k mmq optimizations#80
liangliangchang wants to merge 5 commits into
gfx11from
lichang.q4k-opt

Conversation

@liangliangchang

@liangliangchang liangliangchang commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown

Overview

Test Group TTFT improvement Decode improvement Baseline Candidate >5% regression
Qwen2.5-0.5B-Instruct_Q4_K_M_GGUF_128 standard_128 1.88% -0.03% PASS PASS
Qwen2.5-0.5B-Instruct_Q4_K_M_GGUF_3968 standard_long_context 0.26% -0.26% PASS PASS
Qwen2.5-3B-Instruct_Q4_K_M_GGUF_128 standard_128 5.38% 0.01% PASS PASS
Qwen2.5-3B-Instruct_Q4_K_M_GGUF_3968 standard_long_context 2.06% -0.13% PASS PASS
Qwen2.5-7B-Instruct_Q4_K_M_GGUF_128 standard_128 1.61% 0.46% PASS PASS
Qwen2.5-7B-Instruct_Q4_K_M_GGUF_3968 standard_long_context 0.79% -0.15% PASS PASS
Qwen3-1.7B_Q4_K_M_GGUF_128 standard_128 1.67% -0.03% PASS PASS
Qwen3-1.7B_Q4_K_M_GGUF_3968 standard_long_context 1.09% -0.10% PASS PASS
Qwen3-4B_Q4_K_M_GGUF_128 standard_128 2.39% not rerun PASS PASS
Qwen3-4B_Q4_K_M_GGUF_3968 standard_long_context 1.56% 0.27% PASS PASS
Qwen3-8B_Q4_K_M_GGUF_128 standard_128 1.87% 0.09% PASS PASS
Qwen3-8B_Q4_K_M_GGUF_3968 standard_long_context 0.14% 0.18% PASS PASS
Qwen3.5-4B_Q4_K_M_GGUF_128 standard_128 1.57% 0.03% PASS PASS
Qwen3.5-9B_Q4_K_M_GGUF_128 standard_128 3.42% 0.06% PASS PASS
Llama-2-7B_Q4_K_M_GGUF_128 standard_128 1.56% 3.36% PASS PASS
Llama-2-7B_Q4_K_M_GGUF_1920 standard_long_context 0.37% 0.03% PASS PASS
Gemma-2B_Q4_K_M_GGUF_128 standard_128 2.32% -0.08% PASS PASS
Gemma-2B_Q4_K_M_GGUF_8000 standard_long_context -0.59% 0.01% PASS PASS
SmolLM2-1.7B-Instruct_Q4_K_M_GGUF_128 standard_128 1.18% -0.05% PASS PASS
SmolLM2-1.7B-Instruct_Q4_K_M_GGUF_8000 standard_long_context 0.32% -0.07% PASS PASS
Llama-3.1-8B-Instruct_Q4_K_M_GGUF_128 standard_128 -0.77% 0.19% PASS PASS
Qwen3-30B-A3B-Instruct-2507_Q4_K_M_GGUF_128 standard_128 0.49% not rerun PASS PASS
Gemma-4-E2B-IT_Q4_K_M_GGUF_128 standard_128 2.76% -0.07% PASS PASS
Gemma-4-E2B-IT_Q4_K_M_GGUF_3968 standard_long_context 0.31% 0.06% PASS PASS
Qwen3.5-35B-A3B_Q4_K_M_GGUF_128 standard_128 3.67% not rerun PASS PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_128 standard_128 0.24% not rerun PASS PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_4096 standard_long_context 0.16% -0.02% PASS PASS
Gemma-3-4B-IT_VLM standard_other 0.38% -0.07% PASS PASS
Qwen2.5-VL-3B-Instruct_VLM standard_other 0.30% -0.00% PASS PASS
Qwen2.5-VL-7B-Instruct_VLM standard_other 0.04% -0.01% PASS PASS
Qwen3-VL-4B-Instruct_VLM standard_other 0.41% 0.09% PASS PASS
Gemma-4-26B-A4B-IT_VLM standard_other 0.14% -0.01% PASS PASS
Gemma-4-31B-IT_VLM standard_other 0.68% 0.06% PASS PASS
MiniCPM-V-2_6_VLM standard_other 0.65% 0.06% PASS PASS
MiniCPM-o-2_6_VLM standard_other 0.75% 0.15% PASS PASS
Janus-Pro-1B_VLM standard_other 2.42% -0.00% PASS PASS
Janus-Pro-7B_VLM standard_other 1.43% -0.05% PASS PASS
Qwen3.6-27B_Q4_K_M_GGUF_4096 standard_long_context 0.61% -0.03% PASS PASS
GLM-4.7-Flash_Q4_K_M_GGUF_128 standard_128 -0.15% 0.11% PASS PASS
Gemma-4-12B-it_Q4_K_M_GGUF_128 standard_128 -0.04% -0.03% PASS PASS
Gemma-3-12B-IT_Q4_K_M_GGUF_VLM standard_other 0.81% 0.10% PASS PASS
Gemma-3-12B-IT_Q4_K_M_GGUF_4096 standard_long_context 0.85% 0.10% PASS PASS
Qwen3.5-4B_Q4_0_GGUF_baseline_prose mtp_baseline -0.93% -0.27% PASS PASS
Qwen3.5-4B_Q4_0_GGUF_mtp_prose mtp_verify -0.15% 0.18% PASS PASS
Qwen3.5-4B_Q4_0_GGUF_baseline_code mtp_baseline -1.49% -0.29% PASS PASS
Qwen3.5-4B_Q4_0_GGUF_mtp_code mtp_verify -1.34% 0.30% PASS PASS
Qwen3.5-4B_Q4_0_GGUF_baseline_list mtp_baseline -1.37% -0.02% PASS PASS
Qwen3.5-4B_Q4_0_GGUF_mtp_list mtp_verify -0.89% 0.25% PASS PASS
Qwen3.5-4B_Q4_K_M_GGUF_baseline_prose mtp_baseline -0.95% -0.06% PASS PASS
Qwen3.5-4B_Q4_K_M_GGUF_mtp_prose mtp_verify -1.52% -0.07% PASS PASS
Qwen3.5-4B_Q4_K_M_GGUF_baseline_code mtp_baseline -1.81% -0.06% PASS PASS
Qwen3.5-4B_Q4_K_M_GGUF_mtp_code mtp_verify -1.70% -0.16% PASS PASS
Qwen3.5-4B_Q4_K_M_GGUF_baseline_list mtp_baseline -1.73% -0.17% PASS PASS
Qwen3.5-4B_Q4_K_M_GGUF_mtp_list mtp_verify 0.45% -0.43% PASS PASS
Qwen3.5-9B_Q4_0_GGUF_baseline_prose mtp_baseline -0.25% 0.01% PASS PASS
Qwen3.5-9B_Q4_0_GGUF_mtp_prose mtp_verify 0.01% 0.01% PASS PASS
Qwen3.5-9B_Q4_0_GGUF_baseline_code mtp_baseline 0.86% 0.01% PASS PASS
Qwen3.5-9B_Q4_0_GGUF_mtp_code mtp_verify 0.17% -0.03% PASS PASS
Qwen3.5-9B_Q4_0_GGUF_baseline_list mtp_baseline -0.32% -0.01% PASS PASS
Qwen3.5-9B_Q4_0_GGUF_mtp_list mtp_verify -0.68% 0.30% PASS PASS
Qwen3.5-9B_Q4_K_M_GGUF_baseline_prose mtp_baseline 0.93% 0.04% PASS PASS
Qwen3.5-9B_Q4_K_M_GGUF_mtp_prose mtp_verify 1.09% 0.09% PASS PASS
Qwen3.5-9B_Q4_K_M_GGUF_baseline_code mtp_baseline 1.19% 0.04% PASS PASS
Qwen3.5-9B_Q4_K_M_GGUF_mtp_code mtp_verify 0.38% -0.07% PASS PASS
Qwen3.5-9B_Q4_K_M_GGUF_baseline_list mtp_baseline 0.53% 0.05% PASS PASS
Qwen3.5-9B_Q4_K_M_GGUF_mtp_list mtp_verify -0.65% -0.42% PASS PASS
Qwen3.6-27B_Q4_0_GGUF_baseline_prose mtp_baseline 0.29% 0.01% PASS PASS
Qwen3.6-27B_Q4_0_GGUF_mtp_prose mtp_verify -0.17% 0.04% PASS PASS
Qwen3.6-27B_Q4_0_GGUF_baseline_code mtp_baseline -0.49% -0.01% PASS PASS
Qwen3.6-27B_Q4_0_GGUF_mtp_code mtp_verify 0.94% -0.02% PASS PASS
Qwen3.6-27B_Q4_0_GGUF_baseline_list mtp_baseline 0.21% 0.06% PASS PASS
Qwen3.6-27B_Q4_0_GGUF_mtp_list mtp_verify -0.00% -0.11% PASS PASS
Qwen3.6-27B_Q4_K_M_GGUF_baseline_prose mtp_baseline 0.79% 0.22% PASS PASS
Qwen3.6-27B_Q4_K_M_GGUF_mtp_prose mtp_verify 0.96% -0.37% PASS PASS
Qwen3.6-27B_Q4_K_M_GGUF_baseline_code mtp_baseline 0.09% -0.09% PASS PASS
Qwen3.6-27B_Q4_K_M_GGUF_mtp_code mtp_verify 0.13% -0.43% PASS PASS
Qwen3.6-27B_Q4_K_M_GGUF_baseline_list mtp_baseline 1.31% -0.17% PASS PASS
Qwen3.6-27B_Q4_K_M_GGUF_mtp_list mtp_verify -0.97% -0.13% PASS PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_baseline_prose mtp_baseline 3.48% 0.03% PASS PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_mtp_prose mtp_verify 0.35% -0.12% PASS PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_baseline_code mtp_baseline 1.37% 0.06% PASS PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_mtp_code mtp_verify -0.30% -0.06% PASS PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_baseline_list mtp_baseline 1.30% -0.01% PASS PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_mtp_list mtp_verify 0.31% -0.66% PASS PASS

Additional information

Requirements

liangliangchang and others added 5 commits July 31, 2026 13:18
@liangliangchang @cursoragent
HIP: optimize RDNA3.5 Q4_K MMQ pipeline
2815b3e 
Pipeline Q4_K tile loads with split WMMA dequantization and select the measured tile width by matrix shape.

Assisted-by: GPT-5.6 Sol
Co-authored-by: Cursor <cursoragent@cursor.com>
@liangliangchang @cursoragent
HIP: fix RDNA3.5 Q4_K pipeline integration
a06b975 
Match the refactored 32-row warp layout and limit the specialized J64 path to the small shape where it provides a stable gain.

Assisted-by: GPT-5.6 Sol
Co-authored-by: Cursor <cursoragent@cursor.com>
@liangliangchang @cursoragent
HIP: tune Q4_K wave mapping by tile width
c83bdac 
Use the generic vec-dot with J64 two-row waves and J128 one-row waves, retaining the J64 prefetch loop for the best measured shape coverage.

Assisted-by: GPT-5.6 Sol
Co-authored-by: Cursor <cursoragent@cursor.com>
@liangliangchang @cursoragent
HIP: preserve expert-aware Q4_K MMQ tiling
bec4a25 
Avoid overriding MoE's per-expert tile selection with the dense Q4_K pipeline policy.

Co-authored-by: Cursor <cursoragent@cursor.com>
@liangliangchang @cursoragent
HIP: remove Q4_K MMQ test override
f1cd056 
Keep production dispatch shape-driven and avoid exposing an experimental tile-width environment knob.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@liangliangchang